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ABSTRACT 

This Study outlines and demonstrates a nethodology 
for identifying effective schools using a randoai saaple of student 
data generated by the operation of a school district's Instructional 
Accofliplishaient Inforsiation (lAI) systen. (lAI systeas are designed to 
provide scho^ol districts with inforaation for reviewing and planning 
their instructional programs at the class, school, and district , 
levels.) This study outlines and deaonstrates a methodology for* 
identifying effective schools using a randoa sasqple of sti^ent'data 
generated by the operation of a school district's lAI systea. The 
particular context chosen for demonstration is that of Racially 
Isolated Minority Schools (RIMS) in the Los Angeles Unified School 
District. The study shows that there are substantial instructt1>nal 
accompli shsrants in black RIM schools but that these accompli shunts 
\ are not uniformly spread across the schools. That is, some of the 
black RIM sc^aols are aore effective than others and this study 
identifies the effective subset.. Following the narrative, an'aj^ndix 
discusses technical considerations in using a S percent random, sample 
of. student records in identifying effective schools. (Author/XB) . 
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IDENTIFYING EFFECTIVE SCHOOLS: A CASE STUDY INVOLVING BLACK RACIALLY 
ISOLATED MINORITY SCHOOLS AND INSTRUCTIONAL ACCOMPL I Sfr^lENT INFORMATION 
SYSTEMS 

George Behr and Barry Bachelor 



ABSTRACT 

.instructional Accomplishment Information ( I Al) ^systems are 
designed to provide school districts with information for reviewing 
and planning their instructional -prograiBS. This study outlines and 
demonstrates a methodology for identifying effective schools using a 
randofp sample of student dit^ geiwrated by the operation of a school 
district's lAI system. The particular. context chosen for demonstration 
is that of RacIaUy Isolated Minority Schools (RIMS). The study shows 
that there arc substantial Instructional accomplishments In black RIM 
schools but that they are not tinlformlly sprc«i across tim schools. 
That is, some of the black RIM schools are more effective than others 
and this study identifies tbe effective subset. 
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IDENTIFYING EFFECTtVE SCHOOLS: A CASE STUDY INVOLVING BLACK RACIALLY 
ISOLATED MINORITY SCHOOLS A^JD INSTRUCTIONAL ACCOHPL I SJmEMT INFORMATION 
SYSTEMS 

r 

George Behr and Barry Bachelor 

instructional Accomplishment Information (lAl) systems are designed 
to provide schools and school districts with information for reviewing 
and planning their inst^uctiona1 programs at the class, school, and 
district levels. The l-nnovatlve features of I A I, systems are thoroughly 
discussed by Milazzo and Buchanan (19^). Mllazzo, Buchanan, & Schutz 
(1981) also describe a methodology for reviewing and .planning at the 
district level %#hlle Behr and Yee (1981) describe revla«»ing and planning 
procedures at the class and school level. 

The purpose of this study is to d«TK>n5trate one of the review and 
planning uses at the district level. The particular context chosen for 
demonstration is that of black Racially Isolated Minority Schools (RIMS) 
in the Los Angeles Unified School District. Tte methodology demonstrated 
is one aspect of that described by Milazzo, Buchanan, & Schut* ''981) for 
analysis of district level data bases. 

if 

* 

One of the key principles In the methodology is to Isolate Instruc- 
tional accomplishments and then to track down the school and classroom 
•practices that lead to the identified accon^l ishments. 

The commitment Is to isolating RIMS dtcompl isNnesnts, and not 
just because it is beneficial to point to them— although thit 
%«}u1d be a refreshing change for RIMS. The intention Is to 
isolate RIMS accomplishments so that they can be traced back 
to practices that sctools have some control over. . . . 
The research is for what works In RIMS, to find'it and hang 
onto it, and to genei^allze It to opher RIMS Ip. 7-8]. 



One suggested procedura'. step for, isolating the IfsNnents is 

to "identify those RIHS students who have aco^api ishwent scores that 
are at or above the district average; or^ even better, at or above the 
average fotr non-RIHS students [p. 7]." The purpc»e of this study 
i^ to report the results of j^piylng the suggested procedural step in#* 
identifying RINS accomplishments in the Los- Angeles Unified School 
District based , upon accomplishment information generated and accimiulated 
during'fthe 1979-80 school year. 

Baclcground; I98O Survey of Essential Skills 

Beginning in 1979, the Los Angeles Unified School District in 
collaboration with SWRL has developed, installed and operated an lAI 
system. The systm has several c<M^x>nents which !nc1u(te: 

• Survey of Essential Skills (SES) 

The SES are assessment instruments for Reading, * 
Mathematics and Coi^sition for grades 1, 2, ^ 

5, and 6. They are administered to all 
elementary schrol students each May I Tiwy 
assess students on tfm sicills d^ned essentia? 
For grade-by^-grade pix^res^ in each subject 
area. The assessmsnt ins;trim)ents are supported 
by teacher materials, inservice materials, etc. 

• SES iteportlng System 

The S£S assessment instri»ents are machine 
scannabjie so that SES results car be i}uick1y 
processed and reports prepared for each 
individual stt^ent, for each scNx>1 aggregated 
by grade level witliln scfwot, fqr each admin-* 
' istrative area, and for the district as a 

who1e« The computer software for scanning and 

processing the results is installed chi tfw 

Oistrlct^s oofl^wter and i& €^erated by District 

personiwi. The reporting systi^ is supported * 

by appropriate print and Ii»ervioe materials. 



• SES 5% San^la 

■I 

As part of the computer processingt approxiirately 
S% of the individual student re€X>rds are rarukmly 
selected and stoi^ed in a separate archival file for 
research purposes. Currently for technical reasons 
the remaining SS% of the student records are not 
sav^ed in any readily acrassible form. Each student 
record cxmtains ctonographlc and accompi isNnent 
i nf ormat i on. as fo 1 lows : 



Deffpgraphic Aecowy 1 1 stwien t s 

* ■ . 

1. Attendance tnformatfpn For each subject area — Reading, 

Nathematics, Composition: 

2. Home School Code 

3. Ethnicity 



1. item responses 



2. Skill area scores including 
k. Sex wrltii^ sample scores at 

grades 3 ^nd 6 

5* Language Fluency 

(as applicable for 3. Totfil scores . * 

bilingual program 

considerations) 



identifying RIHS Accon^l ishments in Los todies* 

The SES 5* siw|>le Has used to preliminarily identify those black 
RIM sctoois performing at or above the avera^ for twn-RIM schools . 
For ti^ preliminary analyses, black R I m schools were selected to 
avoid the NES/LES* language considerations involved In the other 
non-black RIM schools. Also, for simpi ici ty white student averages 
t^re used to Identify In aggregate the accoir^l Ishments of the non-RIH 
schools. Thus, the identification logic reduced to Identifying those 
black RIM schools that exceeded In. a spiel flfid-jfashion^the average 
perfoimance of white student-s. ^ substantial number of black RIM 
schools meet such criteria.. ProcedureJ ly^ the Identification process 
has four steps. 



*NES/LES: Non-English Speaking/Limited English Speaking 



The first step In the Identification process Is the calculation of. 
t%«0 kinds of variables fer each grade and subject area . These variables 
are: 

A. The average SES performance score for white students district-wide 

B. The proportion of black students district-wide at or above 
the white student average 

Because there are six grade levels and three subject areas (reading, 
mathematUs, and con^sltlon) at each grade level, 18 A-B pairs of 
variables are calculated. For exafnple, in 3rd grad. reading the district's 
white students' average performance s^re Is 83t correct. Olstrlct-wide, ^ 
29% of the grade -3, black students scored at or above 83* correct. Thus, 
for grade 3, variabte A* Is 83* while variable B Is 29*. 

The second step In the Identification process is to examine each 
black RIM school (enrollment 60* or more black) at each of the 18 points. 
The purpose .is to identify those points where each RIM school has 
•^re-than- Its-share" of black students performing above the nor,-RIM 
(i.e., white) average. Operational I y> 'We- than- Its-share'* is defined 
relative to variable B which Is the distrlct-wlde proportion of black 
.students e.^scccdlng the average performance score for whites. 

Using the previous grade 3 reading exai^le, the logic Is as 
follows- Twenty-nine percent of tte district's grade 3 black students 
exceed the average performance score of 83* correct for grade 3 white 
students In reading. If these particular grade 3 black students are 
equal ly spread among the district's. scNx>ls, then one would expect 
approximately 29*-of the black, grade 3 students In any one school 
to score "Better than Ujmwrect In readti^. H=-^hfy prop ortl er f- hoids 
in each school, then the district -wide proportion would be (and Is) 
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29%. Thus, those schools that have approximately 29? of their black 
grade 3 students scoring above 83* correct arc doing "their share" 
In contributing to, yhe dlstrlct-wi'dc proportion of 29%.* 

r ^ 

However, not all schools will exhibit this «xact percentage; 
• soine will be more and sotne will be less. For exanple, those schools 
showing nwre than 29* of their black, grade 3 students scoring 
above 83% correct In reading aroj doing "incre- than- their- share" in 
contribut^lng to the distrlct^Fde proportion. Thejjs schools will be 
offset by other schools showing less than 29% so hat the overall 
district proportion wf 11 be 23%. • • . * ' 

. Operationally, then, the second step In the identification process 
, is to note whether or not each black RIM school is contributing 'Vnore" 
or "less" to the dist>rict-w?5le proportion Jit each of the l8 possible ^ 
ccMnparison points^ . " 

The third step In the identification process is to look at each 
black RIM school's pattern of acco^l Ishments across the 18 possible 
comparison points . Durational ly, ont easy way to Identify patterns 
is to assign a score froai 0 to 18 to each school. The score Is 
simply the nimber of times the school meets or exceeds each of the 
18 district-wide prdgortlons. The distribution of black RIM schools 
across this kind of point' scale Is shown In Table 1. 

1 

The distribution of scores In Table 1 clearly shm*s that accon^l Ishnwnts 
are not uniformly spread across the black RIM schools. Twenty-nine percent 
of the schools score 9 points or more which ratans that these schools 
^cced the distrlct-wlde pro^ajrtlon at tl« 18 comparison points the 
majority of times. Approximately 45% of the school scores are In 
the 5 to 8 point range. 0y»r^l4, Table ! show? that there arc accom- 
plishments in the black RIM schools. 



' Table 1 

Otstrlbutlon of diack RIN Schools 
Across the ftiint Scale 



Pointy 




Kumber 

of 
Soioois 


Perosnt 

of 
Schools 


Cumulative ' 
Per rant 


t 


10 

It) 




1 


• ''^ 1.5 


1.5 




1 r 




2 


2.9 






1 7 




1 


1.5 


5.9 




12 




w 


5.8 


« 

• 11.7 




1 1 




o 


8.7 


20.^ 




1 A 




2 


2.9 


23.1 




.9 


• 




5,8 


29.1 




• 

ft 
0 




« 

7 


10. 1 • 


3^.2 




7 




6 


8.7 


^7.9 " 




6 




10 


1*1.5 


62.4 




5 




8 


11.6 ' 


74,0 


* 






6 


8.7 


62^7 










5.8 


88.5 








2 


2.9 


91.4 




I 




k 


5.8 


97.2 








2 


2.9 


100.0 




Total 




69 / 


100.0 


• f 
m 





*FIve schools did not contain all six grade levels. For these 
schools, the perce n tage -of ectwl^'yolnts^^ eawied -to total possible 
points Mas calculated and then reconverted to the full point scale. For 
example, a school Mith five grade levels can eiaxlffially earn 15 points 
(5 grades time 3 subject areash If they '^arn'* 9 points, that is 
9/15 of the total possible Mh I ch proportionally converts to ,11 points 
(rounded) on the full point scale. 



The ihformation in Table 1 hs graphically displayed in Figure 1 

t 

Which highlights the fact tRat dccxMi^^lishments are not uniformly spread 
across schools, r Tfra smrothed curve in Figure 1 clearly shows that the 
sch<K> Is Separate thwselves Into three fairly distinct groups\ Not ' 
surprisingly, the bulk of the schools are in the middle gro^> whi^Ie 
the outside groups represent the extr^nes in accOTpl ishments with the 
highest scoring group (the right nKist^ group) being larger than the 
lowest scoring grqup (the left nms't group). 

What accounts for these separatlpns? It is unKkely that 
demographic or ncMi-sctKx>l ing characteristics acovint for nKist of the 
variability. They are all black RIM schools^ Thougfh there* are demo- ^ 
graphic variations between these. It Is a likely possibi 1 1 ty that . 
difference In schooling and instructional practices betw^n the 
scliools account for the separation. \iAiatever the determinants, the 
proposition is empirically verifiable s imp fy Jby, taking a c4oscr look 
(in a subsequ'.nt stiMiy) at these "schools . \ ^ 



NumiMirof 
Sehoolt 




NttfiAor of Prtnls 

( . 



Figure 1. DistrilMJtion Of Black RiM Schools Across The Point Scale 
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Before moving cn^1» establishing criteria to select schools, the 
tot^l score needs to be examined from another perspective. There are 
sirt>patterns of accc^l IsNnnts that don't necessarily lead to a high 

total sdKMl sore. For ex«np1e, a school may .show "s«ires" In all 

" « ■ 

six grades of mattwmatlcs but not s^re In reading or o^iposltlon. 
That Is, Its math^^tlcs s(»re' Is six; Its readli^ srare Is zero; Its 
oon^KisIt Ion score Is 2»ro; therefore. Its overall school score Is six. 
Such a pattern reveals school accomp j ! sHments* In mathematlscs that are 
hidden In an overall jpdiool score. Because the purpose of this Investl" 
gatlpn is to identify instruct I <hi| I accoi^llshm^nts in RIH sdiools, 
such subject area sid»patterns are of definite interest. Accord! ngly, . 
Table. jZ displays the <^served subject ar;^ school patterns of the black 
'RIM scho&ls. 



Though the;*e are some uneveh patterns, t\w scores bet«^n subject 

areas are n»f^ 'insistent than inconsistent. TNs few obvicHis inconsls- 

tencies make mathematics appear to be^-the subject area score that differ 
♦ • 

either hlglier Or Imer, from both reading and opposition. Tt» mathe- 
metlts patterns are extracted amf shMm In Table 3» ^ 



Table' 3 

« 

Mathanatics. Patterns 
' Pattern 



Score . tteading . HatNawatics Coti^sitlon 

12 3 ' 6 ,3 

il . ^'^-<^5 1 .5 

' S \ 3 
8 k '1.3 * 

7 . ' 2 % u 
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Table 2 



SiAJect Mrea Scor« Fattsrm 





taint 




Score Patternft 




Nio^r of 


Scale 


Heading 




Cei^iOtftton 


Patterns 


18 






D 




15 


6 


* e 


5 






S 


J 


i. 




12 


3 




ii 


1 y 


11 












5 < 


1 


c 

3 






. 3 




3 






3 








10 


S 








9 


3 

3 


3 


J \ 

- 2 \ 
2 


* 


8 


• 


1 










3 










2 


* 

k 






- 2 


3 


3 




• ■ 


. J 


2 - 










• 

k 


2 




7 


2 


2 


3 






3 


2 


2 






2 




• 

1 






2" 


1 






S 


2 


it 

2« 


z 






- } 1 


3 


* 2 


2 




2 


* 
1 


3 


2 




•3 


2 


t 


2 


5 


2 


1 


2 


3 




• 2 


2 


1 


z 




3 


1 


i 






I 


1 


3 




k 


1 


1 


2 






2 , 


1 


1 






0" 


2 


2 






1 


3 


0 






1 




1 






1 


0 


2 






2 


c . 






2 


1 • 


t 


0 




1 


0 


m 

1 








0 


0 


1 




0 


0 


0 


0 





Total 



Patterns for the S »chooU not contain! f»9 all 6 grade levels are not Included 

in Table 2. Their raw eo^e patterns JMfed on 5 trades are 3/3/3, 1/1/2, 2/1/0, 
m/0, and 0/0/1. 

- ■ • 13 
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Table k shows the correlations between subject area scores and 
between subject area scores and total scores. The correlations reinforce 
the previously noted points that (1) the within sciiool patterns across 
subject areas are more consistent than Inconsistent and (2) the strongest 
distinction occurs between laatheinatlcs and language arts. 

Table k 
Correlations 



Subject 


R 


M C T 


R 






N 


.59 




C 


..73 


.53 - 


T . 


.89 


.82 .87 - 



R: Readino; M: Hatheoatics; C: Composition: T: Total 

The fourth and final step in the identification process fs to set 
s<me score or pattern criteria ami identify by iwne those schools that 
meet or exceed the criteria. For example, oras criteria could be: 

a. eitiwr a total school score of 9 or more, 

b. or, a score of 4 or more in a subject area 

Nine and four are cut-off points that do identify schools wherein 
the majority of the grade-by-subject-area instruc*-iona1 programs di^^BT 
to be relatively effective on a coo^rlson basis. Schrols meeting the 
criteria «^>pear to have more consistent acompl Ishments across qrade 
levels and/or subject areas than those schools below the cut-off points. 
When the above criteria are ai^lled to the 69 biack RIM schools in Los 
Angeles, 27 sdwols (391) neet these particular criteria. 

& 

There are several points that should be Icept in mind relative to 

the 27 sctoois and the selection c^terla. FJ.^-Kt, thcr- is ^twiyi, 

14 . 



some arbitra-incss Involved in ciit-off scores. There probably Isn't 
much of^a substantive difference between a total school score of 
8 and one of 9. (Oianglng the criteria from 9 to 8 would select 30 
schools instead of 27 schools.) 

« 

Secondly, "selected/not selected" should not be equated with 
"acconvllshroents/no-accorapllshments." There are accomplishments 
in the ncMi-selected schools. The procedure simply ranks the schools 
on specified criteria and selects from the top of the list. Technical 
considerations relative to the ranking process and cut-off points are 
discussed In Appendix 1. •. 

Nevertheless, the results do show that the accomplishments are 
not uniformly distributed across the schools. That Is. the accomplish- 
ments tend to cluster in some schools more than others. 

r 

Sunwiary 

One of the procedural principles suggested by Hilazzo et al. for 
analyzing lAI district level data bases In a RIMS context Is to 
Identify those students who have accomplishments scores at, or above 
the average for non-RIM students. This study applies the procedure 
to black RIM schools in the Los Angeles Unified Schodl District 
relative to the accompl IsNnent Information yielded by the District's 
1980 Survey of Essential Skills. The study demonstrates that the 
procedure Is practical. In jSartlcular, the effort shows that: 

1. a 5% random sample of student records Is adequate 
to preliminarily Identify such schools 

2. the procedure reveals tte many p<Kltlve accoit^Mshn»nts 
of RIM schools 

3. the accomplishments do cluster In some 41IM schools more 
than others (I.e., the accomplishments arc not uniformly 
spread across all RIM schqpls); the procedure Identifies 

• 7.7 black RIM *»cbnols 



isrcspirAifAiimE 
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k, the preceding points tend to su|>port the !dea that schools 
and sch€x>1tng practices do mke a difference 

J- ^jj^-j^-^^ logical step is to track down the schooling 
practices in the identified schools that^ contribute to 
the observed pattern of accon^illshnents 

6. tkwnty-seven is a very manageable number for foi low-up 
efforts 

« 

As noted in the beginning, "the intention is to isolate RIHS 
accooiplishreenits so that they can be traced back to practices that 
schoois'have some control over. . This study demonstrates pro- 
cedures for isolating the accomplishments. Specifically, 27 schools 
are Identified. The next logical step Is to identify the schooling 
practices. that contribute to the observed pattern of accoi^l i shments. 
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/^>pendlx 1 
Posts-crlpt 

Technical Considerations In Using the H Sample 
In the School Identification Proce;5S 

This study preliminarily Identifier 27 black. RIM schools In 
Los Angeles that show certain patterns of Instructional acoMi^l ishments. 
Thp data source for the Identification process Is the 5% r^ndonr saipple 
of student records drawn from the 19») administration of the Survey of 
Essential Ski Vis. Because of certain characteristics of this data 
base, the word "preliminary" Is purposely used when describing the 
result's of the identification process. 

This postscript descrlb-s the reasons for using the adjective 
^iprelimlnary." The ^Jlscussl&i is Included as a postscript because 
the technical qualifications discussed l^reln do not alter the 16gic 
or conclusions of the study. 

The 5% random sample contains 10,666 individual student records 
sanpled from over kkO different schools In the Los Angeles Ihilfied 
School District. A simple division yields the fact that ttere are 
approximately 2k student records per school in the data base. Since 
the Survey of Essential Skills covers grades 1 throu^i 6, another 
simple division yields the fact that there arc apppoxlmately k student 
records per grade level within each school (each student record contains 
results for all three subject areas). Thus, a relatively small N Is 
used In calculating th« proportion of black students who oicced a 
certain score at a certain grade level Ih each school (e.g., exceeding 
B3% correct In reading at grade 3). This a»all N may introduce "error" 
when examining a particular grade le%«l, subject area combination 
(e.g., grade 3 reading) and trying to decide if the school Is above 
or below, a certain percentage. 
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HoMever, for each school there are 18 such decisions to make 
and a score from 0 to 18 to calculate. The score calculation is 
singly tha sun of yeses over 16 yes/iw decisions. Yes, tlw sdK»o1's 
proportion Is above the di$trlct-«ildf prc^rl^on; No, It Is not. 

Though there my 'Errors" fn thie total scores, it is very, 
very unlikely for a school to get a '^Igh*' school score (e.g., 9 to 
18) based im^tly or solely on "errors." High scores most likely 
represent consistent accon^l i shunts and Iwif s<»res most likely 
represent inconsistent acccmplishiwnts.* 

Alternately stated. If the purpose of , the school's score is to 
identify a "true" score for the school, then the small N\and possible 
•'error" at each of the 18 points may be of concern. But,i\^the purpose 
of the school score is to rank schools and then to select the top 
schooVs. in this sltuatlbn, the small Ns In the ca leu Tat I on process 
are of less concern because they will not affect the identification 
of the extremes-''the relatively high scores and the relet Ivel^y Iom 
scores. , 

However, the situation suggests that cauticm be exercised regarding 
judgments aroijnd the cut-off points, (tn any case, caution is a-lways 
advisable In examining cases near cut-off points.) In recognition of 
this situation and the fact that the cases around the cut-off point have 
not been scrutinized over their full, 1001;, data set, the adjective 
"preliminary" Is used to describe the results of the Identification- 
process of this study. . ' 

In following-up any of the Identified schools, one of the substeps 
will be an examination of their full set of SES data. At -that point the 
specific school score can be verified.. The full data. set Is not used 
In tiie present study because the cost of retrieving the full data set 

*ln fact, the chances of obtaining the observed distribution of 
score% If the yes /no decisions at each point are purely random is very 
remote (x2 - 52896; df - 18; p - 0) ' 

18 
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for W schoois Is prohibitive. However, retrieving the full data 
set for approximately 27 schools Is feasible. Though it may also be 
feasible to examine the schools Just below the cut-off point, the 
original 27 schools wilt provide a more than adequate number of 
schools to look at in-depth to identify the schooliryg practices that 
contribute to their success. • . 



in summary, some of the 27 Identified school^ may be de-selected 
after examining their full set of SES data. Also, some f&n schools 
below tiie cut-off point may have been '\*rongly" excluded from the 
selection. To them we apologize. For the above reasons we label the 
list as preliminary. However, the net result of the study is a very 
defensible identification of a very manageable number of black RIH 
sclwols for further In-ctepth stwly. 
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